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ABSTRACT 



This paper describes a study that attempted to ascertain the 
degree of success that undergraduates and graduate students, with varying 
levels of experience using the World Wide Web and Web search engines, and 
without librarian instruction or intervention, had in locating relevant 
material on specific topics furnished by the investigators. Because different 
search engines use dissimilar search protocols and interfaces, potential 
problems paralleling those perceived in CD-ROM and online searching were of 
concern. Results of the study demonstrate that most end-users can effectively 
use at least two search engines without professional assistance. The reasons 
for the contrast in end-users 1 success with World wide Web search engines 
compared with problems found in other types of database searching appear to 
be related to the effectiveness of WWW search engines’ software, which can 
quickly scan a large number of documents and weight the retrieval for 
relevance. While online database and CD-ROM vendors have made proprietary 
search syntaxes standard, Web search engines generally provide for natural 
language searching. The success of users' searches is no longer necessarily 
contingent on a knowledge of search protocols but rather on the ability to 
identify correctly the key concepts for which they are searching. (Contains 
30 references) . (Author/SWC) 
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Abstract: The proliferation of online services and CD-ROM databases offered by many competing 
vendors has led to innumerable products with many different interfaces. Information professionals have 
spent a great deal of time teaching end-users how to use these widely varying interfaces. The World 
Wide Web is similar to its predecessors inasmuch as it is navigable by search engines provided by 
various services. The purpose of this research was to determine if end-users, without the benefit of 
professional instruction , could employ different search engines to locate relevant information on specific 
topics located on the World Wide Web. Because different search engines use dissimilar search protocols 
and interfaces, potential problems paralleling those perceived in CD-ROM and online searching were of 
concern. The results of this study, however, demonstrated that most end-users could effectively use at 
least two search engines without professional assistance. The reasons for the contrast in end-users ' 
success with World Wide Web search engines compared with problems found in other types of 
database searching appear to be related to the effectiveness of WWW search engines’ software, which 
can quickly scan a large number of documents and weight the retrieval for relevance. While online 
database and CD-ROM vendors have made proprietary search syntaxes standard, Web search engines 
generally provide for natural language searching. The success of users’ searches is no longer necessarily 
contingent on a knowledge of search protocols but rather on the ability to identify correctly the key 
concepts for which they are searching. 

Keywords: end-users, natural language, search engines, searching, students, World Wide Web 



1 . Introduction 

World Wide Web search engines, variously called 'navigation services,’ 'Internet directories,’ 'searchable subject 
trees,’ 'WWW databases’ or 'meta-search engines’ are presently the only practical solution for locating specific 
information in cyberspace. Although numerous print resources offer important information concerning the content 
of valuable Web sites, these columns, articles, newsletters and directories are static (Ref 1). The World Wide 
Web’s salient characteristic is its dynamic nature. The Web’s constant evolution renders even the most authori- 
tative site reviews dated almost as quickly as they are published. Search engines are key to unlocking the Web’s 
potential. As information professionals we correctly view ourselves as a cadre possessing the advanced training 
and skills required to assist end-users in performing efficient searches of resources of ail types (including print, 
CD-ROM and online). Having participated in continuing education and workshops, digested articles, read 
monographs and practised our craft for years, we assume that searching is the quintessential blending of art and 
science that invariably requires ingenuity, experience and thesauri. This is unquestionably true when the online 
files being searched emanate from, for instance, STN or Dialog. CD-ROMs from SilverPlatter, University 
Microfilms International and NewsBank, not to mention the dozens of small companies offering compact disc 
databases (each 'contributing’ its own nuances to the searcher/database interaction), require considerable 
mental agility to ensure efficient searching. Professional searchers scans their memories for the answers to 
questions such as 'What is the truncation symbol?’ 'What is the order of processing of operators?’ 'What are the 
correct field qualifiers?’ 

Similarly, many Web search engine sites harbour pages of 'Help,’ 'Hints,’ 'Tips’ or 'FAQs’ (Frequently Asked 
Questions) to facilitate relevant retrieval. End-users, especially novices, are often oblivious to these aids. Can 
end-users expect to locate useful information on the World Wide Web given their untrained approach to 
searching? This experiment provides a starting point for consideration of this topic. 
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2. Literature review and comments from Internet discussion list 
participants 

As Nelson stated in The lonely life of the end-user,’ electronic resources pose significant problems for casual 
database searchers (Ref 2). Friend referred to 'Interface chaos’ (Ref 3). Distress is pervasive among information 
professionals concerned with end-users’ ability to negotiate electronic resources (Refs 4,5,6). Although many 
articles have discussed training information professionals to use the World Wide Web for research and reference 
purposes (Refs 7,8,9), few have considered training students (Refs 10,11). Brandt covered teaching end-users to 
search the World Wide Web from a conceptual perspective (Ref 12). He also authored practical articles 
addressing instruction in the use of the Web through a text-browser called Lynx (Refs 13,14). 

The investigators found no published papers addressing the more specific issue of teaching end-users to 
interact with Web search engines. A question posted on the discussion lists WEB4LIB (discussion of WWW in 
libraries) and Bl-L (Bibliographic instruction discussion group), in which an investigator requested anecdotal 
accounts of librarians’ perceptions of students’ search engine usage, drew a significant number of interesting 
responses. The following overview of responses reinforces the impression that end-users seldom use search 
engines effectively. One information specialist commented, 'My consistent experience has been that students are 
using the search engines the way they use the card catalogue — haphazardly until they find something that 
works. When they finally ask for help, they are amazed at the hits I am (usually) able to generate and I see them 
use that information in future searches. What I’m finding with the Internet is that it’s taking a lot of one-on-one 
time to teach students how to search with various engines’ (Ref 15). A related remark stated, 'In my observation 
they [end-users] use Yahoo extensively and the other tools in the most basic and limited way. And do not seem 
to question exhaustive lists of irrelevant results’ (Ref 16). Jessamyn West, Technology Project Coordinator for the 
City of Seattle (Washington, USA), a role that makes her a primary planner of teaching children and parents how 
to use computers and the Internet, stated, 'I think there still is the stigma of folks thinking the Internet is for play 
and therefore a dearth of instruction techniques’ (Ref 17). To balance this misconception, she has a Web site 
document aimed to assist students in using search engines (Ref 18). Another declared, 'This is a special interest 
area of mine ... With the different interfaces I stress that people should be checking the help screens so that they 
can take full advantage of the searching capabilities of the search engine. And of course it is important to remind 
them that the search engines are not mutually exclusive. I recommend using a minimum of three’ (Ref 19). Several 
additional respondents affirmed that they felt making end-users aware of the existence of Web search engines 
and of their proper usage was essential; they took the opportunity to edify in both formal bibliographic instruction 
formats as well as at point of use (Ref 20). 

Search engines have been acclaimed as the panacea for end-user meandering (Ref 21). Engine providers 
imply that their services are user-friendly. For example, the top of the Lycos form presently reads, Search the 
Web for’ followed by a fill-in box. If accessing Lycos via Netscape Navigator 2.01 ’s NetSearch function, the Lycos 
form reads 'Find exactly what I want’ followed by a fill-in box. The implication is that merely entering words will 
yield relevant information. InfoSeek’s president and chief executive Robin Johnson takes a more explicit 
perspective. Commenting on a recent advertising campaign promoting his company’s services, he said, 'The 
point of the advertising is that you don’t have to be an information retrieval specialist because InfoSeek is like 
your personal information concierge’ (Ref 22). 

Information professionals, however, remain concerned that as with CD-ROMs, end-users are not optimally 
deploying the Web search engines. Is this concern warranted? Herb White commented that for end-users 
searching is a means to an end, not an end in itself. End-users simply want an answer (Ref 23). The investigators’ 
a priori hypothesis was if end-users have historically resisted learning the eccentricities of searching online 
databases and CD-ROMs, they would probably perform unimpressively using the Web’s search engines. 



3. Purpose 

This experiment attempted to ascertain the degree of success that undergraduates and graduate students, with 
varying levels of experience using the World Wide Web and Web search engines, and without librarian instruction 
or intervention, had in locating relevant material on specific topics furnished by the investigators. 



4. Methodology 

4.1 . Selection of participants 

The university’s electronic bulletin board was used to recruit subjects for the experiment. This ensured that the 
subjects had some baseline knowledge of networks, electronic mail and computers (i.e. they were reading the 
notice and replying via computer). Individuals with extensive experience were rejected (e.g. the university’s 
Webmaster had asked to participate and was not allowed). Thirty students requested information on the project. 
Ten students were recruited and completed the experiment. 
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4.2. Administration of participant survey and participant orientation 

Each participant first completed a brief survey describing her or his previous experience using both traditional 
electronic library resources, as well as her or his previous experience using World Wide Web search engines. 
Students were advised of the following: 'A World Wide Web search engine generally refers to a searchable 
database of files available on the World Wide Web; depending on the coverage and strength of the search engine 
and the manner in which a search is entered, relevant items may be found at various locations on the World Wide 
Web. Hundreds of search engines exist; for the purposes of this experiment you will search the following engines: 
Open Text Index and AltaVista.’ 

Participants were also told that although the engines basically performed the same function, each may appear 
different and have its own special features. Investigators stated, 'It is up to you to try to discover some of those 
features and employ them to do a better search.’ No other instruction or elucidation was provided. 

Each participant was given seven topics to search. The seven topics were identical for each participant. 
Investigators mentioned that for each topic valid information existed on the World Wide Web. The investigators 
began the searching portion of the experiment by having the participant use the 'Open Text Index.’ Students were 
instructed to try one topic at a time. They were further instructed to browse the results and revise their search 
strategy if they felt it was necessary. Because the investigators were interested in high relevance rather than high 
recall, the participants were asked to print only the first ten 'hits’ from the search they were most satisfied with 
for each topic. They were reminded not to print the pages of any specific Web site; using an overhead trans- 
parency, the investigators illustrated exactly what information should be printed. If zero 'hits’ were retrieved, the 
subjects were instructed to print the page that stated this fact in lieu of a list of sites. 

Investigators met with each student individually. Each had a two hour time limit for the appointment. The first 
half hour was occupied by the orientation and survey. The remaining one and one-half hours could be used for 
searching. 

4.3. Participant searching 

Each student was allocated a maximum of 45 minutes to search seven topics on the 'Open Text Index.’ 
Investigators than used a bookmark to switch participants to AltaVista. The same seven topics were searched. 
Students were again allowed to search for 45 minutes. 

Printouts were collected and retained with each student’s survey. Data was gathered from the surveys and 
the search results. Using the printouts, the investigators inspected the students’ search strategies as well as their 
final results. 

4.4. Investigator searching 

In preparation for the experiment, the investigators attempted to search numerous academic or popular topics 
sought on the World Wide Web using both engines. Seven topics were chosen. Each topic was adequately 
covered by both AltaVista and the Open Text Index. Students’ retrieval would be compared with 'benchmark’ 
retrieval located by the investigators. The investigators used the available special search features that each 
engine offered whenever necessary or appropriate to enhance retrieval. 

4.5. Success criteria 

Investigators determined that a student’s search could be deemed successful if they had retrieved the 
benchmark site(s) that was(were) being used as the reference point(s), or if the retrieval included a site with equiv- 
alent or otherwise useful information on the topic, or if the site retrieved contained a hypertext link to one of the 
benchmark sites. 



5. Search topics 

The search topics consisted of: 

(1) locate the full text of the Magna Carta; 

(2) locate a listing of Royal Shakespeare Company productions; 

(3) locate Supreme Court Decisions — but not Canadian; 

(4) locate information on a press conference given in Madrid by John Major concerning the European Union; 

(5) locate a site containing pictures from the Hubble Space Telescope; 

(6) locate free stock quotes; 

(7) locate information on Poet’s Corner in London’s Westminster Abbey. 
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5.1 . Benchmark sites and search success scoring 

As of the time this paper was written the following sites were established as the 'benchmark sites’ (i.e. those sites 
that had the best information on the topic). Other sites that participants located that had equivalent information 
were scored as though they were benchmark sites. Sites that mirrored the benchmark sites or had hypertext links 
to the benchmark sites were scored as benchmark sites. Sites retrieved that had incomplete or only moderately 
useful material were viewed by the investigators as 'marginal sites’ and were scored as being one-half the value 
of the benchmark sites. If a participant failed to retrieve any sites, found only irrelevant sites or did not complete 
a search for a topic, a zero was recorded. 'Benchmark sites' were worth two points. 'Marginal sites’ were worth 
one point. All other retrieval or lack thereof was worth zero points. 

Full text of the Magna Carta 

http://www.let.rug.nl/~welling/usa/magna.html 

http://www.traveler.com/~hrweb/legal/magnacrt.html 

http://www.cs.indiana.edu/statecraft/magna-carta.html or equivalent sites, etc. 

Royal Shakespeare Company productions 

http://www.uktw.co.uk/rsc.html http://www.hiway.co.uk/~ei/rsc.html or equivalent sites, etc. 

Supreme Court Decisions — but not Canadian 

http://www.law.cornell.edu/supct/ 

http://www.hplus.harvard.edu/alpha/supcrt.html or equivalent sites, etc. 

John Major's Press Conference on the European Union in Madrid 

http://www.fco.gov.uk/current/1995/dec/16/major_pressconference_madrid-summit.txt or equivalent sites, etc. 

Hubble Space Telescope pictures 

http://www.physics.carleton.ca/research/Hubble or equivalent sites, etc. 

Free Stock Quotes 

http://www.westergaard.com:8080/goquote.html 
http://www.li.net/~stanh/stocks.html or equivalent sites, etc. 

Poet's Corner at Westminster Abbey 

http://www.demon.co.uk/webguides/london/tourist/to_visit/list.html 

http://trinket.crocker.com/living/berkeley/london/chapter2/westm.html or equivalent sites, etc. 



6. Results 

Two points was the maximum score for a successful search. Seven topics were assigned and they were to be 
searched on two engines. A perfect score equalled twenty-eight points. 

6.1 . Participants scores compared to self-report of computer use/search 
experience 

All but one participant had used Microsoft Windows. Five participants reported using the World Wide Web 'once 
in awhile,’ four had no experience and one used it frequently.’ Out of the five that used the Web once in awhile’ 
two had found the search engines frustrating while three reported 'sometimes find what I want.’ Table 1 shows 
the scores for all participants as well as their responses to the survey questions. It is interesting to note that the 
second lowest score was made by a participant who claimed to 'usually find what I want’ and stated 'frequently’ 
using the World Wide Web and the search engines. 
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Table 1: Participants’ responses to survey. 



Subject 


1 have used 

Microsoft 

Windows 


1 have used 

the library’s 

traditional 

electronic 

databases 

(CD-ROM, 

OPAC, etc.) 


1 have used 
the World 
Wide Web 


1 have used 
World Wide 
Web “search 
engines” 
(Yahoo, 
Infoseek, etc.) 


To date, my 
experience 
using World 
Wide Web 
search 
engines has 
usually been: 


Seach 
Success 
Score note: 
max score 
possible = 28 
(Success % 
in brackets 
cf. Table 2) 


A grad student 


5 


1 


3 


3 


3 


19 (68%) 


B undergrad 


2 


4 


1 


1 


1 


22 (79%) 


C grad student 


3 


2 


4 


4 


4 


16 (57%) 


D undergrad 


5 


4 


3 


3 


3 


26 (93%) 


E undergrad 


4 


2 


3 


2 


3 


22 (79%) 


F undergrad 


1 


3 


1 


1 


1 


20 (71 %) 


G undergrad 


4 


3 


1 


1 


1 


23 (82%) 


H undergrad 


4 


4 


3 


3 


3 


20 (71%) 


1 undergrad 


3 


2 


3 


2 


2 


23 (82%) 


J grad student 


3 


4 


1 


1 


1 


14 (50%) 



Survey Questions 

1 . I have used Microsoft Windows: 

1 = Never, 2 = Rarely, 3 = Once in a while, 4 = Frequently, 5 = Several times each week 

2. I have used the library’s traditional electronic databases (CD-ROMs, OPACs): 

1 = Never, 2 = Rarely, 3 = Once in a while, 4 = Frequently, 5 = Several times each week 

3. I have used the World Wide Web: 

1 = Never, 2 = Rarely, 3 = Once in a while, 4 = Frequently, 5 = Several times each week 

4. I have used World Wide Web “search engines” (Yahoo, Infoseek, etc.): 

1 = Never, 2 = Rarely, 3 = Once in a while, 4 = Frequently, 5 = Several times each week 

5. To date, my experience using World Wide Web search engines has usually been: 

1 = I have no experience, 2 = Frustrating, 3 = I sometimes find what I want, 4 = I usually find what I want, 
5 = I always find what I want 



6.2. Web search engine disparity not a significant factor in results 

The differences and similarities of the Open Text Index and AltaVista have been reported (Ref 24). Both provide 
excellent search capabilities but neither is mutually exclusive with regard to content or search features. Both 
engines, however, listed the benchmark sites. AltaVista’s search protocol is closer to the traditional online or CD- 
ROM databases. It allows truncation (not automatically but optionally). It allows nested searching. AltaVista’s 
online documentation is comprehensive. But having inspected the printouts, the investigators determined that 
participants of this study did not capitalise on most of these advantages. Open Text Index is a powerful engine 
that ignores no stop words. It is extremely straightforward in its presentation. 

For either engine, the possible total points that could be scored if all participants had a perfect score on every 
topic was 140. Participants scored a total of 105 points using AltaVista; a 75% total success rate. They scored 
100 total points using Open Text Index; a 71% total success rate. 

Ironically, participants who mentioned a preference liked Open Text Index’s interface compared to AltaVista’s. 
Although these engines’ search screens are dissimilar and have unique features, this was not a significant factor 
in relevant retrieval. 



7. Discussion and conclusions 

Experience or lack of experience with the World Wide Web or its search engines did not seem to affect the majority 
of participants’ performance. Lack of instruction or unfamiliarity with search protocols generally had a negligible 
affect on the overall outcome of the end-users’ efforts (Table 2). Search engine appearance and disparities in the 
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search engine’s forms had little effect. Notwithstanding the small sample size, the results indicate that end-users 
of varying experience could use Web search engines reasonably well. The lowest total score was produced by 
Subject J who accrued fourteen points. Subject J reported having never used the Web and consequently had no 
experience using search engines. Earning 16 total points, Subject C had the next lowest score. Ironically Subject 
C reported frequently using the Web, including frequently using its search engines, and 'I usually find what I want.’ 
Aside from these two scores, participants scored acceptably or well, particularly considering they had no 
instruction and, as evidenced by the printouts, had not used many advanced techniques. 

Table 2: Distribution of successful searches. 



Percentile 


Frequency 


Scores 


0 


0 


n/a 


10th 


0 


n/a 


20th 


0 


n/a 


30th 


0 


n/a 


40th 


0 


n/a 


50th 


2 


14,16 


60th 


1 


19 


70th 


4 


20,20,22,22 


80th 


2 


23,23 


90th 


1 


26 


100th 


0 


n/a 



Distribution of Sucevtaful Search#* 




On the WEB4LIB discussion list Paul Hollands, Internet Information Officer at Loughborough University, 
commented that The Holy Grail of electronic information retrieval is precision natural language searching.’ He 
qualified this, however, by remarking, 'Unfortunately, having tested Natural Language engines in my previous job 
which could handle questions like “Give me reports about airbag crash tests that failed,” we still found that 
standard Boolean searching gave much greater precision’ (Ref 25). Quint has written that it is natural for end- 
users to want to search without applying difficult command languages; natural language searching must be 
realised if end-users are to have the best possible access to information (Ref 26). Within the context of this exper- 
iment, natural language searching apparently works as well, in many cases, as the more precise methods that 
the investigators employed. The scores of the student participants in this experiment bear this out. Students who 
invoked any of the hints provided by the search engines were in the minority. The search aid most used was 
AltaVista’s asterisk as a truncation symbol. Professional searchers familiar with AltaVista will recall that with each 
new search a different 'tip’ is posted at the top of the AltaVista page. The tips on the use of the asterisk, use of 
quotation marks for entering phrases and the use of the + (plus) sign to restrict retrieval are frequently rotated. 
Otherwise students simply picked out the main concepts of the topic and entered them. In several cases, unfor- 
tunately, students did not select enough keywords. For example, in some cases students merely entered John 
Major’ and clicked the 'Search’ button. The investigators had assigned a very specific topic and this strategy was 
far too broad. 

Notwithstanding the overall acceptable outcome of these searches, deficiency in formulating search 
strategies is a subject warranting comment. In the experimental searches students were provided topics that 
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were articulated in precise terms. Even this advantage proved insufficient to guide some of their attempts. 
Omitting keywords (e.g. Westminster, Madrid, European Union) was an apparent problem. Typographical/spelling 
errors (e.g. Westminister, Abby) hampered retrieval but were never caught or corrected. Consider the student with 
an imperfect understanding of a topic. In this situation many students may falter in locating useful information. 
Oberman peripherally supports this: 'A growing number of studies of students and their interactions with 
electronic databases clearly illustrates the dismal abilities of students, even at the most basic level, of being able 
to match their subject needs with appropriate computer retrieval systems’ (Ref 27). Her comment reflects the idea 
that the search instruction process must begin at a much more basic level than some information professionals 
may assume. 

Other authors have stated that end-users continue to desire the information specialist’s intervention when 
searching complex topics (Refs 28,29). These authors, however, have not focused their comments on end-users 
searching the World Wide Web. Many end-users have mounted their own home pages. The Web is available in 
many homes. Students explore the Web without trepidation. While the Web may be called a database, it is 
certainly not a database in the same way Chemical Abstracts, Medline, Psyclnfo or the electronic Science 
Citation Index are databases. 

We recognise the dynamic nature of the World Wide Web. Its contents constantly change. (Open Text Index 
altered its 'Simple Search’ form midway through this experiment!) P. Steven Thomas, Coordinator of Library 
Instruction at Illinois State University (Normal, Illinois, USA) stated during the 'teaching students Web search 
engines’ thread on the WEB4LIB discussion list: 'Search engines are evolving as we speak ... Expert systems are 
going to increasingly take over the online search logic many of us have learned through training and experience. 
These systems aren t going to replace us (i.e. librarians) but they will greatly simplify the search complexities that 
have characterised mediated online searching in the past’ (Ref 30). As information professionals consider 
teaching end-users to manipulate Web search engines efficiently, perhaps only the basics require emphasis. End- 
users should be apprised of the valuable information contained in the hints, help, tips and FAQs that are available 
for many engines. Information professionals more wisely may choose as their primary goals the satisfactory 
formulation and refinement of search queries along with the need to choose sufficient, valid keywords. This being 
done, many end-users may need be led no further than Netscape Navigator’s 'NetSearch’ button or to a site 
containing links to Web search engines. Compared with earlier electronic information formats, for example CD- 
ROMs and 'value-added’ online databases offered by various vendors, the efficient nature of World Wide Web 
search software, with its ability to handle natural language queries, is apparent. 
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